Image editing and repair

ABSTRACT

A method for healing a target region on an input image is described. A preview image is received; the preview image may reflect a down-sampled image of an original image. The method determines a target region for the preview image. The target region indicates a segment of the preview image designated for healing. The method may then heal the target region associated with the preview image using a transformation. The method may store one or more parameters associated with the healed preview image. The method may then provide for display the healed preview image to a user on a mobile device.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/090,476, filed Apr. 4, 2016 and titled “Image Editing andRepair,” which claims priority under 35 U.S.C. § 119(e) to U.S.Provisional Patent Application No. 62/144,814, filed Apr. 8, 2015 andtitled “Image Editing and Repair,” both of which are incorporated byreference in their entirety.

BACKGROUND

Image healing is the process of synthesizing pixels to replace a givenset of pixels in an image. Image healing is a well-established imageenhancement application for image processing because it allows users toremove artifacts in an image. However, it is difficult to use certainconventional image enhancement techniques on mobile devices with smallscreen sizes due to the limited space available in preview images onsuch small screens. These conventional techniques thus often make itnecessary to revisit edits made on a mobile device on a device with alarger screen.

SUMMARY

The system and method relate to editing and modifying images. Inparticular, the system and method relate to image healing. Morespecifically, the system and method relate to editing and modifyingimages by healing artifacts from an image.

According to one aspect of the subject matter described in thisdisclosure, a method for healing a target region on an input image isdisclosed. The method includes receiving, using one or more computingdevices, image data from a client, the image data including parametersassociated with an input image; computing, using the one or morecomputing devices, a scaling of the parameters of the image data basedon a predetermined scaling factor; determining, using the one or morecomputing devices, a first transformation for a target region associatedwith the input image; responsive to determining the first transformationfor the target region associated with the input image, updating, usingthe one or more computing devices, the input image based on thetransformation; and providing, using the one or more computing devices,the updated input image for display to a user.

In general, another aspect of the subject matter described in thisdisclosure may be embodied in methods that include receiving a firstinput from the user for previewing the input image; providing fordisplay a preview image associated with the input image; receiving asecond input from the user for a target region associated with the imagepreview, the target region indicating a segment of the input image to bemodified; modifying the segment of the input image; providing fordisplay the modified input image to the user; that the preview imageincludes a down-sampled image of the input image; identifying atransformation for a search region on the preview image, the searchregion including the transformation and the target region; determiningboundary parameters of the target region and boundary parameters of thesearch region; analyzing the preview image for boundary parametersimilarities between the boundary parameters of the target region andthe boundary parameters of the search region; computing a boundarydeviation-error for the boundary parameter similarities between thetarget region and the search region, the boundary deviation-errorreflecting a similarity confidence between the boundary parameters ofthe target region and the boundary parameters of the search region;transforming a set of pixels from the search region on the targetregion; that transforming the set of pixels from the search region onthe target is responsive to the boundary deviation-error satisfying apredetermined pixel threshold; computing a target region refinementusing the scaled parameters and seamless cloning; and computing a borderrefinement for the boundary parameters of the target region using imageenhancement.

Other aspects include corresponding methods, systems, apparatus,non-transitory computer readable medium, and computer program products.

BRIEF DESCRIPTION OF THE DRAWINGS

The specification is illustrated by way of example, and not by way oflimitation in the figures of the accompanying drawings in which likereference numerals are used to refer to similar elements.

FIG. 1 is a block diagram illustrating an example system for healing animage.

FIG. 2 is a block diagram illustrating an example computing device.

FIG. 3 is a flowchart illustrating an example method for healing animage.

FIG. 4 is a flowchart illustrating an example method for computingborder refinement on a healed image using transformation.

FIGS. 5A-5C are graphical illustrations of healing a target region on aninput image.

FIGS. 6A-6B are example user interfaces depicting a user specifiedtarget region for healing on an input image.

FIGS. 7A-7B are additional example user interfaces depicting a userspecified target region for healing on an input image.

DETAILED DESCRIPTION

In one implementation, the system and method edit and modify an image,and specifically perform image healing. User devices (e.g., a cellularphone, a tablet, a computer, etc.) suffer from a technical problem inthat they tend to have limited processing power and are not alwaysconnected to the Internet with a high speed connection. When the userdevice is connected to the Internet with a low bandwidth connection, theimages shared over the low bandwidth Internet connection requiresminimizing the size of the file uploaded or downloaded, rather thanapplying the edits to a full resolution image version because of thelimited computing capabilities of the user device and the availablebandwidth for sending and receiving the image. For example, if a userrequests to edit a 40-megabyte (40 MB) image, the 40 MB file isminimized to 4 MB for transfer to the user device, rather thantransferring the 40 MB file. The user edits the 4 MB file on the userdevice and then transfers that file back to the server. However, theedits made on the 4 MB file may become distorted and altered when thefile is increased back to its original 40 MB file size. Thoseconventional techniques required the user to apply edits on a fullresolution version of the image before sharing or uploading, which isnot possible with low bandwidth Internet connection typical of computingdevices.

The present implementation discussed herein provides a technicalsolution to the issue of using image enhancement applications on a userdevice and further provides users the ability to share the edited imagewith other users in a user association or an online service, forexample, on a social network site by determining scaling parameters ofan image and determining a transformation for a target region of theimage. By using scaling and transformations, images may be minimized andsent to the user device for editing and then up scaled back to fullresolution without distorting the image edits.

FIG. 1 illustrates a block diagram of an example system 100 for healingan image. In the depicted implementation, the system 100 includes animage server 134 and user devices 106 a through 106 n. In the depictedimplementation, the devices 134 and 106 are electronicallycommunicatively coupled via a network 102. However, the presentdisclosure is not limited to this configuration and a variety ofdifferent system environments and configurations may be employed and arewithin the scope of the present disclosure. Other implementations mayinclude additional or fewer computing devices, services, and/ornetworks.

It should be recognized that in FIG. 1 as well as the other figures usedto illustrate an implementation, an indication of a letter after areference number or numeral, for example, “106 a” is a specificreference to the element or component that is designated by thatparticular reference numeral. In the event a reference numeral appearsin the text without a letter following it, for example, “106,” it shouldbe recognized that such is a general reference to differentimplementations of the element or component bearing that generalreference numeral.

In some implementations, the devices of the system 100 may use acloud-based architecture where one or more computer functions orroutines are performed by remote computing systems and devices at therequest of a local computing device. For example, a user device 106 maybe a computing device having hardware and/or software resources and mayaccess hardware and/or software resources provided across the network102 by other computing devices and resources, including, for instance,other user devices 106 and the image server 134, or any other devices ofthe system 100.

The network 102 may be a conventional type, wired or wireless, and mayhave numerous different configurations including a star configuration,token ring configuration or other configurations. Furthermore, thenetwork 102 may include a local area network (LAN), a wide area network(WAN) (e.g., the Internet), and/or other interconnected data pathsacross which multiple devices may communicate. In some implementations,the network 102 may be a peer-to-peer network. The network 102 may alsobe coupled to or include portions of a telecommunications network forsending data in a variety of different communication protocols. In someother implementations, the network 102 includes Bluetooth communicationnetworks or a cellular communications network for sending and receivingdata including via short messaging service (SMS), multimedia messagingservice (MMS), hypertext transfer protocol (HTTP), direct dataconnection, wireless application protocol (WAP), email, etc. Inaddition, although FIG. 1 illustrates a single network 102 coupled tothe user devices 106 that are illustrated and the image server 134, inpractice one or more networks 102 may be connected to these devices.

In some implementations, the user device 106 (any or all of 106 athrough 106 n) are computing devices having data processing and datacommunication capabilities. In the illustrated implementation, the users114 a through 114 n interact with the user device 106 a and 106 n, viasignal lines 112 a through 112 n, respectively. The user devices 106 athrough 106 n are communicatively coupled to the network 102 via signallines 104 a through 104 n respectively. Although two user devices 106are illustrated, the disclosure applies to a system architecture havingany number of user devices 106 available to any number of users 114.

In some implementations, a user device 106 includes a workstationcomputer, a desktop computer, a laptop computer, a netbook computer, atablet computer, a smartphone, a set-top box/unit, an InternetProtocol-connected smart TV including a computer processor capable ofreceiving viewer input, accessing video content on computer networks(e.g., the Internet), and executing software routines to provideenhanced functionality and interactivity to viewers, or the like. Insome implementations, the user device 106 may be a handheld wirelesscomputing device which may be capable of sending and receiving voiceand/or data communications.

The user device 106 may include a computer processor, a memory, a powersource, and a communication unit including one or more networkinterfaces for interacting with the network 102, including, for example,wireless transceivers to broadcast and receive network data via radiosignals. The user device 106 may also include one or more of a graphicsprocessor; a high-resolution touchscreen; a physical keyboard; forwardand rear facing cameras; a Bluetooth® module; memory storing applicablefirmware; and various physical connection interfaces (e.g., USB, HDMI,headset jack, etc.); etc.

Additionally, an operating system for managing the hardware andresources of the user device 106, application programming interfaces(APIs) for providing applications access to the hardware and resources,a user interface module for generating and displaying interfaces foruser interaction and input, and applications including, for example,applications for web browsing, capturing digital video and/or images,etc., may be stored and operable on the user device 106. While FIG. 1illustrates two or more user devices 106, the present disclosure appliesto any system architecture having any number of user devices 106.

In the depicted implementation, the user devices 106 a through 106 ncontain a user application 108 (illustrated as 108 a through 108 n)executable by a processor (not shown) of the user device 106 to providefor user interaction, and to send and receive data via the network 102.In particular, the user application 108 is operable to instruct the userdevice 106 to render user interfaces, receive user input, and sendinformation to and receive information from the image server 134, andthe other components of the system 100. In these or otherimplementations, the user application 108 may be stored in memory (notshown) of the user device 106 and is accessible and executable by aprocessor (not shown). In further implementations, the user device 106may include an image healer engine 136. The user 114 (114 a through 114n) utilizes the user application 108 to exchange information with theimage healer engine 136, and the image server 134, as appropriate toaccomplish the operations of the present implementations.

The image server 134 may be a computing device that includes aprocessor, a memory, and network communication capabilities. The imageserver 134 is coupled to the network 102, via a signal line 132. Theimage server 134 may be configured to obtain a plurality of images forhealing from the user device 106 and/or other components of system 100,via the network 102. Although one image server 134 is shown, it shouldbe understood that multiple servers may be utilized, either in adistributed architecture or otherwise. For the purpose of thisapplication, the system configuration and operations performed by thesystem are described in the context of a single image server 134.

In some implementations, the image server 134 comprises an image healerengine 136 for image healing. The image healer engine 136 may receive auser input from the user application 108, and then heal an image basedon the received user input. For example, the user 114 may provide inputindicating a target region (e.g. an unwanted area for a final image) ofan image to be healed; the image healer engine 136 may analyze the imageand heal the image based on the user input. By way of another example,the user 114 may provide input indicating multiple target regions of animage-frame associated with a video clip to be healed; the image healerengine 136 may analyze each target region in the image-frameindividually and/or collectively and heal the plurality of targetregions based on the user input.

As depicted in FIG. 1, the image healer engine 136 is shown in dottedlines to indicate that the operations performed by the image healerengine 136 as described herein may be performed either server-side(e.g., image server 134) or user-side (e.g., user devices 106 a through106 n), or a combination of the two. Additional structure, acts, and/orfunctionality of the image healer engine 136 is described in furtherdetail below with respect to at least FIG. 2.

FIG. 2 is a block diagram of an example computing device 200, which maybe representative of a computing device included in the image server 134and/or the user device 106. As depicted, the computing device 200 mayinclude a processor 216, a memory 218, a communication unit 220, a datastore 222, and one or more of a user application 108, and an imagehealer engine 136, which may be communicatively coupled by acommunication bus 214.

Depending upon the configuration, the computing device 200 may includedifferent components. For instance, in a server-side implementation, thecomputing device 200 may include the image healer engine 136. However,in an example client-side implementation, the computing device 200 mayinclude the user application 108, and/or the image healer engine 136. Itshould be understood that the above configurations are provided by wayof example and numerous further configurations are contemplated andpossible.

The bus 214 may include a communication bus for transferring databetween components of a computing device or between computing devices, anetwork bus system including the network 102 or portions thereof, aprocessor mesh, a combination thereof, etc. In some implementations, theuser application 108 and the image healer engine 136 may cooperate andcommunicate via a software communication mechanism implemented inassociation with the bus 214. The software communication mechanism mayinclude and/or facilitate, for example, inter-process communication,local function or procedure calls, remote procedure calls, network-basedcommunication, secure communication, etc.

The processor 216 may execute software instructions by performingvarious input, logical, and/or mathematical operations. The processor216 may have various computing architectures to method data signalsincluding, for example, a complex instruction set computer (CISC)architecture, a reduced instruction set computer (RISC) architecture,and/or an architecture implementing a combination of instruction sets.The processor 216 may be physical and/or virtual, and may include asingle core or plurality of processing units and/or cores. In someimplementations, the processor 216 may be capable of generating andproviding electronic display signals to a display device, supporting thedisplay of images, capturing and transmitting images, performing complextasks including various types of feature extraction and sampling, etc.In some implementations, the processor 216 may be coupled to the memory218 via the bus 214 to access data and instructions therefrom and storedata therein. The bus 214 may couple the processor 216 to the othercomponents of the computing device 200 including, for example, thememory 218, communication unit 220, and the data store 222.

The memory 218 may store and provide access to data to the othercomponents of the computing device 200. In some implementations, thememory 218 may store instructions and/or data that may be executed bythe processor 216. The memory 218 is also capable of storing otherinstructions and data, including, for example, an operating system,hardware drivers, other software applications, databases, etc. Thememory 218 may be coupled to the bus 214 for communication with theprocessor 216 and the other components of the computing device 200.

The memory 218 may include a non-transitory computer-usable (e.g.,readable, writeable, etc.) medium, which can be any non-transitoryapparatus or device that can contain, store, communicate, propagate ortransport instructions, data, computer programs, software, code,routines, etc., for processing by or in connection with the processor216. In some implementations, the memory 218 may include one or more ofvolatile memory and non-volatile memory (e.g., RAM, ROM, hard disk,optical disk, etc.). It should be understood that the memory 218 may bea single device or may include multiple types of devices andconfigurations.

The communication unit 220 may include one or more interface devices forwired and wireless connectivity with the network 102 and the othercomponents and/or components of the system 100 including, for example,the user devices 106, the image server 134, and the data store 222, etc.For instance, the communication unit 220 may include, but is not limitedto, CAT-type interfaces; wireless transceivers for sending and receivingsignals using Wi-Fi; Bluetooth, cellular communications, etc.; USBinterfaces; various combinations thereof; etc. The communication unit220 may be coupled to the network 102 via the signal lines 104 and 132.In some implementations, the communication unit 220 can link theprocessor 216 to the network 102, which may in turn be coupled to otherprocessing systems. The communication unit 220 can provide otherconnections to the network 102 and to other components of the system 100using various standard communication protocols, including, for example,those discussed elsewhere herein.

The data store 222 is an information source for storing and providingaccess to data. In some implementations, the data store 222 may becoupled to the components 216, 218, 220, 108, and/or 136 of thecomputing device 200 via the bus 214 to receive and provide access todata. In some implementations, the data store 222 may store datareceived from the other devices 106 and/or 134 of the system 100, andprovide data access to these devices. The data store 222 may include oneor more non-transitory computer-readable mediums for storing the data.In some implementations, the data store 222 may be incorporated with thememory 218 or may be distinct therefrom. In some implementations, thedata store 222 may include a database management system (DBMS). Forexample, the DBMS could include a structured query language (SQL) DBMS,a NoSQL DMBS, various combinations thereof, etc. In someimplementations, the DBMS may store data in multi-dimensional tablescomprised of rows and columns, and manipulate, e.g., insert, query,update and/or delete, rows of data using programmatic operations.

The image healer engine 136 is software, code, logic, or routines forediting or modifying an image by healing the image. As depicted in FIG.2, the image healer engine 136 may include a user interface module 202,an image analysis module 204, a refinement module 206, and a synthesismodule 208. It should be understood that the refinement module 206 asindicated by the dotted line in FIG. 2 is optional and may not berequired at all times during the operations and/or functionalityperformed by the image healer engine 136 as described elsewhere herein.

In the depicted implementation, the components 202, 204, 206, and/or 208are electronically communicatively coupled for cooperation andcommunication with each other, the user application 108, the processor216, the memory 218, the communication unit 220, and/or the data store222. These components 202, 204, 206, and 208 are also coupled forcommunication with the other components (e.g. user device 106) of thesystem 100 via the network 102.

In some implementations, the user interface module 202, image analysismodule 204, refinement module 206, and the synthesis module 208 are setsof instructions executable by the processor 216, or logic included inone or more customized processors, to provide their respectivefunctionalities. In some implementations, the user interface module 202,image analysis module 204, refinement module 206, and the synthesismodule 208 are stored in the memory 218 of the image server 134 and areaccessible and executable by the processor 216 to provide theirrespective functionalities. In any of these implementations, the userinterface module 202, image analysis module 204, refinement module 206,and/or the synthesis module 208 are adapted for cooperation andcommunication with the processor 216 and other components of the imageserver 134.

The user interface module 202 is software, code, logic, or routines forreceiving inputs from a user. In some implementations, the inputsreceived from the user include a request for a preview image. Forexample, a user may launch an image enhancement application on the userdevice 106 and select an image to preview. The preview image may reflecta down-sampled image of an input image. In other implementations, theuser interface module 202 receives an input from the user indicating atarget region for healing on the preview image. For example, the targetregion may reflect artifacts (e.g. dust spots) present in the previewimage that the user desires to remove. In some implementations, the userinterface module 202 receives image data from a client, the image dataincluding parameters associated with an input image.

In some implementations, the user interface module 202 provides aplurality of predetermined healing contours in addition to the previewimage. The predetermined healing contours may include a plurality oftarget region dimensions for a user to apply as a target region forhealing on the preview image. Examples, of the predetermined healingcontours, may include, but are not limited to, a square, a triangle, acircle, and/or any other geometric configuration etc. In someimplementations, the user interface module 202 receives a user-definedhealing contour for a preview image. For example, a user may open apreview image in the user application 108 and sketch a particular shapeon the preview image, the particular shape indicating a target regionfor healing. In some implementations, the user interface module 202 maysend the received inputs to the image analysis module 204, which maythen use those inputs to perform its acts and/or functionalitiesthereon.

The image analysis module 204 is software, code, logic, or routines fordetermining a target region on a preview image and qualitatively healingthe target region associated with the preview image using atransformation. In some implementations, the image analysis module 204receives a target region for a preview image from the user interfacemodule 202. For example, the target region may indicate a segment of thepreview image for healing. The segment of the preview image may includean artifact for example, but is not limited to, a dust spot, an objectin the preview image, a face of a person, a casted shadow, etc. In someimplementations, the image analysis module may receive a first inputfrom the user interface module 202, the first input indicative of a userrequest for previewing an input image on a user device. In someimplementations, the image analysis module 204 receives a second inputfrom the user for a target region associated with the image preview fromthe user interface module 202. The image analysis module may modify thetarget region and/or a segment of the input image based on the targetregion.

In some implementations, a target region on an input image is determinedautomatically. For instance, the image analysis module 204 may analyzean input image and identify one or more classifying characteristics ofthe input image. The image analysis module 204 may then retrieve apredetermined source region from memory 218 and/or data store 222, thepredetermined source region sharing similar classifying characteristicsas the input image.

In some implementations, the image analysis module 204 determines atransformation for a target region associated with an input image usinga plurality of scaled parameters and seamless image cloning process. Insome implementations, a seamless image cloning process may be a methodof image editing incorporating graph-cut segmentations or mean-valuecoordinates to interpolate the membrane of a target portion of an imagebeing edited as discussed elsewhere herein. The image analysis module204 may heal the input image based on the transformation.

In some implementations, the image analysis module 204 may receive aplurality of target regions for a preview image from the user interfacemodule 202 and/or any other components of computing device 200. Thepreview image may include a down-sampled image of an input image. Insome implementations, the image analysis module 204 receives image datafrom a client, the image data including parameters associated with theinput image. The image analysis module 204 may then compute a scaling ofthe parameters of the image data based on a predetermined scalingfactor.

In some implementations, a preview image may be a full resolutionversion of an input image. A target region may include dimensions m×n,the m×n dimensions of the target region reflecting pixels that areassociated with intensity values. In some implementations, responsive todetermining the target region on the preview image, the image analysismodule 204 may analyze the preview image for a plurality of candidatesource regions. For example, the image analysis module 204 may analyzepixel values associated with the target region and identify a pluralityof candidate source regions in the preview image that have similar pixelvalues as the target region. In some implementations, a candidate sourceregion may be transformed onto a target region responsive to similarpixel values of a candidate source region meeting a predetermined pixelthreshold associated with the target region. In some implementations,the image analysis module 204 may initialize pixels of a target regionand assign an offset to the target region; the image analysis module 204may then recursively search a concentric radius in a preview image todetermine pixel similarity between the target region and a candidatesource region.

In some implementations, the image analysis module 204 searches apreview image for a plurality of candidate source regions based onfeatures in the preview image, and a candidate source region of theplurality of candidate source regions may be identified and transformedonto a target region. The image analysis module 204 may communicatecooperatively with one or more other components of computing device 200via bus 214, for computing and comparing features associated with atarget region and one or more candidate source regions. Examples offeatures associated with the target region, may include, but are notlimited to, image colors in YCbCr and a local variance of a 7×7neighborhood associated with an input Luma. The YcbCr is indicative of afamily of color spaces associated with a preview image, Y being a lumacomponent (e.g. brightness in an image) and components Cb and Cr beingblue and red difference components in the image. In someimplementations, the image analysis module 204 may store the computedfeatures in memory 218 and/or data store 222.

In some implementations, the image analysis module 204 synthesizespixels to replace a given set of pixel for a target region in an image(e.g. preview image). For instance, the image analysis module 204determines a pixel neighborhood for an input image, and the imageanalysis module 204 may initialize a synthesized pixel patch from atarget region. In some implementations, the synthesized patch may be apredefined pixel kernel. The image analysis module 204 may analyze theinput image for non-value pixels that border value-pixels. The imageanalysis module 204 may then determine a set of patches in the inputimage that have similarities with the non-value pixels' valued-pixelsneighbors and replace the target region with the synthesized pixels. Insome implementations, the image analysis module 204 synthesizes pixelsfor an under-sampled input image.

In some implementations, the image analysis module 204 qualitativelyheals a target region on the preview image using a transformation. Insome implementations, the image analysis module 204 and/or the imageserver 134 may retrieve a scaling factor associated with an image (e.g.down-sampled image displayed on a user device). In some implementations,the image analysis module 204 may search for a candidate source regionusing a transformation. The transformation may include a CovarianceMatrix Adaptation Evolution Strategy (CMA-ES) method. For example, theCMA-ES performs a random sampling from a bimodal Gaussian distributionand evolution of the preview image. In some implementations, the imageanalysis module 204 may update the bimodal Gaussian distribution forfuture target regions that may include similar features associated withhistorical target regions assigned to one or more other images.

In some implementations, the image analysis module 204 may receive oneor more restriction regions on a preview image that indicate regionsthat are excluded from qualifying as a candidate source region. Forexample, the one or more restriction regions on the preview imageindicate regions that are excluded from qualifying as a candidate sourceregion. The restriction regions may reflect an artifact or imperfection(e.g. pimple, blemish on a user's face) in the preview image that theuser wishes not to be considered as a replacement for a target region.In some implementations, responsive to identifying a candidate sourceregion in a preview image, the image analysis module 204 may transformthe candidate source region onto a target region associated with thepreview image.

In some implementations, the image analysis module 204 may store one ormore parameters associated with a healed preview image in memory 218and/or data store 222. An example of one or more parameters of a healedimage, may include, but is not limited to, a candidate source region, atransformation associated with the candidate source region and a targetregion, and/or dimensions of a preview image and candidate sourceregion. In some implementations, the acts and/or functionalitiesperformed by the image analysis module may be performed server-side viaimage server 134. The image analysis module 204 may transmit one or moreparameters associated with a preview image and/or healed preview imagethat reflects a down-sampled version of an input image; the image server134 may apply the one or more parameters associated with thedown-sampled version to perform the acts and/or functionalities on afull-sampled image. In some implementations, the image analysis module204 may send the healed target region associated with a preview imageand one or more parameters of the healed image to the refinement module206 and/or the synthesis module 208, which may then use those inputs toperform its acts and/or functionalities thereon.

The refinement module 206 is software, code, logic, or routines formodifying a mask and/or membrane of a candidate source region whencloning the candidate source region onto a target region associated withan image (e.g. preview image). In some implementations, the refinementmodule 206 receives a candidate source region for a target region on apreview image from the image analysis module 204. The refinement module206 may identify a transformation for a search region on a previewimage, the search region including the transformation and the targetregion on the preview image.

In some implementations, the refinement module 206 determines boundaryparameters of a target region and boundary parameters of a searchregion. The refinement module 206 may qualitatively analyze the previewimage for boundary parameter similarities between the boundaryparameters of the target region and the boundary parameters of thesearch region. The boundary parameters associated with a target regionmay be user-parameterized. In some implementations, the refinementmodule 206 may compute a boundary deviation-error for the determinedboundary similarities between the target region and the search region.The boundary deviation-error may reflect a similarity confidence betweenthe boundary parameters of the target region and the boundary parametersof the search region.

In some implementations, the refinement module 206 may compute a targetregion refinement using seamless cloning process. For instance, therefinement module 206 may retrieve predetermined boundary parameters fora target region from memory 218 and/or data store 222 and compute arefinement of the predetermined boundary parameter using any cloningprocess that is well known in the art. In some implementations, therefinement module 206 synthesizes pixels to replace a given set ofpixels for a target region in an image (e.g. preview image).

In some implementations, the refinement module 206 may compute boundaryparameter refinements of a target region and/or a search region using ashortest path algorithm. An example of a shortest path algorithm, mayinclude, but is not limited to, a Dijkstra's Algorithm and/or imageenhancement method. For example, the refinement module 206 may utilizethe Dijkstra's algorithm to determine a path with a lowest cost in acontour around a target region and/or search region in an image (e.g.preview image). In some implementations, a lowest cost in a contour isdefined as a sum of gradients along a path of an error (e.g. differencebetween candidate source region and target region) in addition to a sumof an error along the gradients. In some implementations, the refinementmodule 206 may compute boundary parameter refinements utilizingconvolution pyramids. In some implementations, the refinement module 206preserves image quality of a down-sampled image by applying convolutionpyramids to an original image (e.g. full resolution image) and/or anyother known image-processing configuration performed on it to preservedetails in the original image.

In some implementations, the refinement module 206 performs refinementsof boundary parameters associated with a target region, search region,and/or candidate source region. In some implementations, refinement ofboundary parameters provides a healed image with less visible seamsaround the contour of the target region, search region, and/or candidatesource region. In other implementations, the refinement module 206 maysend the refined boundary parameters associated with a healed image tothe synthesis module 208 and/or any other components of system 100,which may then use those inputs to perform its acts and/orfunctionalities thereon.

The synthesis module 208 is software, code, logic, or routines forproviding for display a healed image to a user on a user device. In someimplementations, the synthesis module 208 may receive a healed imagefrom the image analysis module 204 and provide the healed image fordisplay to the user via user interface module 202 on the userapplication 108. In other implementations, the synthesis module 208 mayreceive the healed image from the refinement module 206 and provide thehealed image for display to the user via user interface module 202 onthe user application 108. In further implementations, the synthesismodule 208 may transmit the healed image to one or more other componentsof system 100 to perform their acts and/or functionalities thereon.

Methods

FIG. 3 is a flowchart illustrating an example method for performingimage healing. The method 300 begins by receiving 302 a preview image,the preview image reflecting a down-sampled image of an original image(e.g. input image). In some implementations, the operations in block 302may be performed by the user application 108 and the down-sampled imagemay be down-sampled by the image healer engine 136 on the image server134 and sent from the image server 134 to the user device 106 via thenetwork 102. In some implementations, the down-sampled image may begenerated by the user application 108 a on the user device 106 a andsent to another user device 106 n via the network 102.

The method 300 continues by determining 304 a target region for thepreview image, the target region indicating a segment of the previewimage for healing. In some implementations, the operations in block 304may be performed by the image analysis module 204. In someimplementations, the image analysis module 204 automatically determinesa target region for the preview image using standard image processingtechniques discussed elsewhere herein to identify a discrepancy relatedto an artifact in the preview image and classifying a segment of thepreview image including the discrepancy as the target region. In someimplementations, the image analysis module 204 receives a user definedtarget region for a preview image from the user interface module 202based on user input, and upon receiving the user defined target region,the image analysis module 204 may determine a target region on thepreview image based on the user defined target region. For example, thetarget region may indicate a segment of the preview image for healing.The segment of the preview image may include an artifact for example,but not limited to, a dust spot, an object in the preview image, aperson's face, a casted shadow, etc.

The method 300 continues by healing 306 the target region associatedwith the preview image using a transformation. In some implementations,the operations in block 306 may be performed by the image analysismodule 204. In some implementations, the image analysis module 204 maysearch for a candidate source region using a transformation. In oneimplementation, the search may be implemented using the CovarianceMatrix Adaptation Evolution Strategy (CMA-ES) method. The candidatesource region may be used by the image analysis module 204 for acomparison of pixel values between the target region and the candidatesource region. Based on the comparison, the image analysis module 204may heal pixels included in the target region by altering the pixels toresemble the candidate source region, as discussed elsewhere herein.

The method 300 may then continue by storing 308 one or more parametersassociated with the healed preview image. In some implementations, theoperations in block 308 may be performed by the image analysis module204. The image analysis module 204 may store the one or more parametersin memory 218 and/or data store 222 for future application on one ormore different preview images sharing similar features, pixel values,contours, etc. Once the method 300 completes storing the one or moreparameters, the method 300 may then provide 310 for display the healedimage to a user. In some implementations, the operations in block 310may be performed by the synthesis module 208. For instance, thesynthesis module 208 may receive a healed image from the image analysismodule 204 and provide the healed image for display to the user on theuser application 108. In some implementations, healing of a targetregion may be multi-layered. Multi-layered healing includes healingdifferent portions of the target region with different layers. Eachdifferent layer may apply a specific healing to correct the portion ofthe target region. Then the multiple different layer may be applied tothe target region of an input image or combined and applied to thetarget region of an input image. In some implementations, a user may beable to reverse one or more sub-layers of the multi-layered healedtarget regions and undo the healing of the target region.

FIG. 4 is a flowchart illustrating an example method for computingborder refinement on a healed image using a transformation. The method400 begins by identifying 402 a transformation for a search region on apreview image, the search region including the transformation and atarget region. In some implementations, the transformation for thesearch region may be a scaling factor applied to the portion of theimage including the search region and the search region may be a portionof the preview image that includes the target region as well as asurrounding area near the target region to be healed. In someimplementations, the operation performed in block 402 may be performedby the refinement module 206. For instance, the refinement module 206may identify a transformation for a search region on a preview image bysearching the preview image for one or more candidate source regions. Insome implementations, the candidate source regions may be portions ofthe preview image showing with characteristics for the healed portion ofthe image to match. In some implementations, the candidate sourceregions may be data models of what a healed region may look like and therefinement module 206 may use the data models to transform the targetregion.

The method 400 may continue by determining 404 boundary parameters ofthe target region and boundary parameters of the search region. In someimplementations, the operation performed in block 404 may be performedby the refinement module 206. For instance, the refinement module 206may determine boundary parameters by analyzing one or morecharacteristics of the target region and/or the search region. Anexample of a characteristic of the target region and/or the searchregion may include, but is not limited to, pixel values associated witha contour of the target region and/or the search region.

The method 400 may continue by analyzing 406 the preview image forboundary parameter similarities between the boundary parameters of thetarget region and the boundary parameters of the search region. In someimplementations, the boundary similarities may be a value of how similarpixels included in a target region are compared to pixels included inthe search region. For example, in some implementations parameters suchas color, intensity or texture between the boundary and surroundingareas can be analyzed for similarity. In some implementations, theoperation performed in block 406 may be performed by the refinementmodule 206. The method 400 may continue by computing 408 a boundarydeviation-error for the boundary parameter similarities between thetarget region and the search region. For instance, the boundarydeviation-error may reflect a confidence measure for the similaritybetween the boundary parameters of the target region and the boundaryparameters of the search region.

The method 400 may continue by computing 410 a target region refinementusing seamless image cloning. In some implementations, the operation inblock 410 may be performed by the refinement module 206. For instance,the refinement module 206 may retrieve predetermined boundary parametersfor a target region from memory 218 and/or data store 222 and compute arefinement of the predetermined boundary parameter using any cloningmethod that is well known in the art. In some implementations, theoperation in block 410 is optional and may not be part of the method400.

The method 400 may continue by computing 412 a border refinement forboundary parameters associated with the target region using animage-enhancing method. In some implementations, the border refinementmay be healing the boundary of the target region to blend with the restof the input image. In some implementations, the operation performed inblock 412 may be performed by the refinement module 206. For example,the refinement module 206 may utilize a shortest path algorithm todetermine a path with a lowest cost in a contour around a target regionand/or search region in an image (e.g. preview image).

The method 400 may continue by transforming 414 pixels from the searchregion onto the target region. In some implementations, the operation inblock 414 may be performed by the refinement module 206. In someimplementations, the refinement module 206 transforms pixels from thesearch region onto the target region, responsive to the boundarydeviation-error satisfying a predetermined pixel threshold.

FIGS. 5A-5C are graphical illustrations of healing a target region on aninput image. In particular, interface indicated by reference numeral 500depicts an input image 500 provided to a user for editing on a userapplication. The input image 500 may include an object 502 (e.g. targetregion) that a user wishes to remove from the image. For instance,reference numeral 502 indicates that the user has selected a windowassociated with the input image 500 for removal. FIG. 5B depicts a mask504 associated with the target region selected by a user, shown in FIG.5A. In some implementations, the refinement module 206 computes a maskfor a target region selected by a user in an input image. FIG. 5Cdepicts a healed version 506 of the input image 500. Also, as evidencedby reference numeral 508 depicting the healed target region, the healedtarget region no longer includes the selected window as shown in FIG.5A.

FIGS. 6A-6B are example user interfaces depicting a user specifiedtarget region for healing on an input image. Referring to FIG. 6A, auser interface 600 includes a target region 602 a, a target contourselection pane 604, and a guidance zoom overlay window 606. As depictedin FIG. 6A, user interface 600 displays an image 608 a (e.g. previewimage) with a target region 602 a indicating an object (e.g. segment) ofthe image 608 a for healing. The target contour selection pane 604 mayinclude a predefined square-target-region, a predefinedcircular-target-region, and a predefined triangular-target-region.

Referring to FIG. 6B, a user interface 600 includes a healed targetregion 602 b, a target contour selection pane 604, and a guidance zoomoverlay window 606. As depicted in FIG. 6b , user interface 600 displaysa healed image 608 b (e.g. healed preview image) with a healed targetregion 602 b. As evidenced with reference numeral 602 b, the objectpreviously shown in FIG. 6A under reference numeral 602 a is not presentin the healed image 608 b of FIG. 6B.

FIGS. 7A-7B are additional examples of user interfaces depicting a userspecified target region for healing on an input image. Referring to FIG.7A, a user interface 700 includes a target region 702 a, a targetcontour selection pane 704, and a guidance zoom overlay window 706. Asdepicted in FIG. 7A, user interface 700 displays an image 708 a (e.g.preview image) with a target region 702 a indicating an object (e.g.segment) of the image 708 a for healing. For instance, the target region702 a includes graffiti on a door.

Referring to FIG. 7B, a user interface 700 includes a healed targetregion 702 b, and a guidance zoom overlay window 706. As depicted inFIG. 7b , user interface 700 displays a healed image 708 b (e.g. healedpreview image) with a healed target region 702 b. As evidenced withreference numeral 702 b, the object previously shown in FIG. 7A underreference numeral 702 a is not present in the healed image 708 b of FIG.7B.

In the foregoing description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the specification. It will be apparent, however, to oneskilled in the art that the technology can be practiced without thesespecific details. In other implementations, structures and devices areshown in block diagram form in order to avoid obscuring the description.For example, the specification is described in some implementationsabove with reference to user interfaces and particular hardware.However, the description applies to any type of computing device thatcan receive data and commands, and any peripheral devices providingservices.

The systems and methods discussed herein do not require collection orusage of user personal information. In situations in which certainimplementations discussed herein may collect or use personal informationabout users (e.g., user data, information about a user's social network,user's location, user's biometric information, user's activities anddemographic information), users are provided with one or moreopportunities to control whether the personal information is collected,whether the personal information is stored, whether the personalinformation is used, and how the information is collected about theuser, stored and used. That is, the systems and methods discussed hereincollect, store and/or use user personal information only upon receivingexplicit authorization from the relevant users to do so. In addition,certain data may be treated in one or more ways before it is stored orused so that personally identifiable information is removed. As oneexample, a user's identity may be treated so that no personallyidentifiable information can be determined. As another example, a user'sgeographic location may be generalized to a larger region so that theuser's particular location cannot be determined.

Reference in the specification to “some implementations” or “animplementation” means that a particular feature, structure, orcharacteristic described in connection with the implementation isincluded in at least some instances of the description. The appearancesof the phrase “in some implementations” in various places in thespecification are not necessarily all referring to the sameimplementation.

Some portions of the detailed description are presented in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The specification also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may include a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, and magnetic disks,read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, flash memories including USB keyswith non-volatile memory or any type of media suitable for storingelectronic instructions, each coupled to a computer system bus.

The specification can take the form of an entirely hardwareimplementation, an entirely software implementation or implementationscontaining both hardware and software elements. In some implementations,the specification is implemented in software, which includes but is notlimited to firmware, resident software, microcode, etc.

Furthermore, the description can take the form of a computer programproduct accessible from a computer-usable or computer-readable mediumproviding program code for use by or in connection with a computer orany instruction execution system. For the purposes of this description,a computer-usable or computer readable medium can be any apparatus thatcan contain, store, communicate, propagate, or transport the program foruse by or in connection with the instruction execution system,apparatus, or device.

A data processing system suitable for storing and/or executing programcode will include at least one processor coupled directly or indirectlyto memory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers.

Network adapters may also be coupled to the system to enable the dataprocessing system to become coupled to other data processing systems orremote printers or social network data stores through interveningprivate or public networks. Modems, cable modems and Ethernet cards arejust a few of the currently available types of network adapters.

Finally, the processes, methods, and displays presented herein are notinherently related to any particular computer or other apparatus.Various general-purpose systems may be used with programs in accordancewith the teachings herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps. The requiredstructure for a variety of these systems will appear from thedescription below. In addition, the specification is not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the specification as described herein.

The foregoing description of the implementations of the specificationhas been presented for the purposes of illustration and description. Itis not intended to be exhaustive or to limit the specification to theprecise form disclosed. Many modifications and variations are possiblein light of the above teaching. It is intended that the scope of thedisclosure be limited not by this detailed description, but rather bythe claims of this application. As will be understood by those familiarwith the art, the specification may be embodied in other specific formswithout departing from the spirit or essential characteristics thereof.Likewise, the particular naming and division of the modules, routines,features, attributes, methodologies and other aspects are not mandatoryor significant, and the mechanisms that implement the specification orits features may have different names, divisions and/or formats.Furthermore, as will be apparent to one of ordinary skill in therelevant art, the modules, routines, features, attributes, methodologiesand other aspects of the disclosure can be implemented as software,hardware, firmware or any combination of the three. Also, wherever acomponent, an example of which is a module, of the specification isimplemented as software, the component can be implemented as astandalone program, as part of a larger program, as a plurality ofseparate programs, as a statically or dynamically linked library, as akernel loadable module, as a device driver, and/or in every and anyother way known now or in the future to those of ordinary skill in theart of computer programming. Additionally, the disclosure is in no waylimited to implementation in any specific programming language, or forany specific operating system or environment. Accordingly, thedisclosure is intended to be illustrative, but not limiting, of thescope of the specification, which is set forth in the following claims.

What is claimed is:
 1. A computer-implemented method comprising:providing, using one or more computing devices, for display, adown-sampled preview image generated from an input image; identifying atarget region of the preview image using the one or more computingdevices, the target region indicating a segment of the preview image tobe modified; responsive to identifying the target region: identifying asource region from a plurality of candidate source regions in thepreview image based on one or more features of the preview image;transforming the source region onto the target region using atransformation to create a healed preview image, wherein thetransformation is based on the source region; wherein transforming thesource region onto the target region includes: comparing pixel values ofthe target region and the source region; and healing one or more pixelsof the target region by altering the one or more pixels based on thesource region; and storing one or more parameters associated with thetransformation and the healed preview image; and causing an update,using the one or more computing devices, of the input image based on theone or more parameters such that one or more image edits associated withthe transformation are not distorted.
 2. The computer-implemented methodof claim 1, wherein the one or more parameters include at least one of:the source region, the transformation associated with the source regionand the target region, or dimensions of the preview image.
 3. Thecomputer-implemented method of claim 1, wherein identifying the sourceregion includes identifying the plurality of candidate source regions ashaving similar pixel values as the target region and identifying thesource region as one of the candidate source regions that has similarpixel values that meet a predetermined pixel threshold associated withthe target region.
 4. The computer-implemented method of claim 1,wherein the one or more features of the preview image include at leastone of: colors or brightness.
 5. The computer-implemented method ofclaim 1, wherein identifying the source region includes searching forthe source region using a second transformation.
 6. Thecomputer-implemented method of claim 1, wherein healing the one or morepixels of the target region is performed by a user device of the one ormore computing devices, and causing the update of the input imageincludes transmitting the parameters by the user device to a server thatperforms the update of the input image.
 7. The computer-implementedmethod of claim 1, wherein the transformation is determined using theplurality of candidate source regions as data models of a healed region,wherein transforming the source region onto the target region isperformed using the data models.
 8. The computer-implemented method ofclaim 1, wherein transforming the source region onto the target regionincludes refining the target region based on boundaries of the sourceregion and the target region.
 9. The computer-implemented method ofclaim 1, wherein one or more restriction regions are defined in thepreview image, wherein the plurality of the candidate source regionsexclude the one or more restriction regions.
 10. Thecomputer-implemented method of claim 1, wherein identifying the targetregion includes identifying a discrepancy related to an artifact in thepreview image and classifying a segment of the preview image thatincludes the discrepancy as the target region.
 11. Thecomputer-implemented method of claim 1, wherein identifying the targetregion includes receiving a definition of the target region via userinput.
 12. A system comprising: a processor; and a memory coupled to theprocessor, the memory storing instructions that, when executed, causethe processor to perform operations comprising: providing for display, adown-sampled preview image generated from an input image; identifying atarget region of the preview image, the target region indicating asegment of the down-sampled preview image to be modified; responsive toidentifying the target region: identifying a source region from aplurality of candidate source regions in the preview image based on oneor more features of the preview image; transforming the source regiononto a target region using a transformation to create a healed previewimage, wherein the transformation is based on the source region; whereinthe operation of transforming the source region onto the target regionincludes operations of: comparing pixel values of the target region andthe source region; and healing one or more pixels of the target regionby altering the one or more pixels based on the source region; andstoring one or more parameters associated with the transformation andthe healed preview image; and causing an update of the input image basedon the one or more parameters such that one or more image editsassociated with the transformation are not distorted.
 13. The system ofclaim 12, wherein the one or more parameters include at least one of:the source region, the transformation associated with the source regionand the target region, or dimensions of the preview image.
 14. Thesystem of claim 12, wherein the operation of identifying the sourceregion includes identifying the plurality of candidate source regions ashaving similar pixel values as the target region and identifying thesource region as one of the candidate source regions that has similarpixel values that meet a predetermined pixel threshold associated withthe target region.
 15. The system of claim 12, wherein the processorperforming the operations is a client device and the operation ofcausing the update of the input image includes transmitting theparameters to a server that performs the update of the input image. 16.The system of claim 12, wherein the operation of transforming the sourceregion onto the target region includes an operation of refining thetarget region based on boundaries of the source region and the targetregion.
 17. The system of claim 12, wherein the operation of identifyingthe target region includes an operation of identifying a discrepancyrelated to an artifact in the preview image and classifying a segment ofthe preview image including the discrepancy as the target region. 18.The system of claim 12, wherein the operation of identifying the targetregion includes an operation of receiving a definition of the targetregion via user input.
 19. A non-transitory computer readable mediumincluding a computer readable program, wherein the computer readableprogram when executed on a computer causes the computer to performoperations comprising: providing for display, a down-sampled previewimage generated from an input image; identifying a target region of thepreview image, the target region indicating a segment of thedown-sampled preview image to be modified; responsive to identifying thetarget region: identifying a source region from a plurality of candidatesource regions in the preview image based on one or more features of thepreview image; transforming the source region onto a target region usinga transformation to create a healed preview image, wherein thetransformation is based on the source region; wherein the operation oftransforming the source region onto the target region includes:comparing pixel values of the target region and the source region; andhealing one or more pixels of the target region by altering the one ormore pixels based on the source region; and storing one or moreparameters associated with the transformation and the healed previewimage; and causing an update of the input image based on the one or moreparameters such that one or more image edits associated with thetransformation are not distorted.
 20. The non-transitory computerreadable medium of claim 19, wherein the computer performing theoperations is a client device and the operation of causing the update ofthe input image includes transmitting the parameters to a server thatperforms the update of the input image.